{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "1QiCFLer1FIe" }, "source": [ "**Lab 13 – Deploy**" ] }, { "cell_type": "markdown", "metadata": { "id": "vCyq3-8y1FIj" }, "source": [ "_This notebook contains the sample from https://github.com/ageron/handson-ml2/blob/master/19_training_and_deploying_at_scale.ipynb, https://github.com/bentoml/BentoML, https://github.com/streamlit/streamlit, https://github.com/gradio-app/gradio and https://evidentlyai.com/blog/tutorial-1-model-analytics-in-production" ] }, { "cell_type": "code", "source": [ "" ], "metadata": { "id": "gz_CRD0BDLKq" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "9J5g6PDs1FIk" }, "source": [ "\n", " \n", " \n", "
\n", " \"Open\n", " \n", " \n", "
" ] }, { "cell_type": "code", "source": [ "!pip install bentoml --pre -qq\n", "!pip install pyngrok -qq\n", "!pip install PyYAML -U -qq\n", "!pip install streamlit -qq\n", "!pip install gradio -qq\n", "!pip install evidently" ], "metadata": { "id": "8YFCNVkuZ3U4", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "4ad88ff5-70de-4f19-8b01-5c834bdf029c" }, "execution_count": 1, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\u001b[K |████████████████████████████████| 602 kB 38.5 MB/s \n", "\u001b[K |████████████████████████████████| 47 kB 6.5 MB/s \n", "\u001b[K |████████████████████████████████| 231 kB 59.7 MB/s \n", "\u001b[K |████████████████████████████████| 53 kB 2.8 MB/s \n", "\u001b[K |████████████████████████████████| 135 kB 75.5 MB/s \n", "\u001b[K |████████████████████████████████| 1.1 MB 30.0 MB/s \n", "\u001b[K |████████████████████████████████| 64 kB 4.5 MB/s \n", "\u001b[K |████████████████████████████████| 182 kB 72.7 MB/s \n", "\u001b[K |████████████████████████████████| 57 kB 6.3 MB/s \n", "\u001b[K |████████████████████████████████| 61 kB 599 kB/s \n", "\u001b[K |████████████████████████████████| 271 kB 66.9 MB/s \n", "\u001b[K |████████████████████████████████| 94 kB 3.8 MB/s \n", "\u001b[K |████████████████████████████████| 144 kB 54.3 MB/s \n", "\u001b[K |████████████████████████████████| 51 kB 8.4 MB/s \n", "\u001b[K |████████████████████████████████| 80 kB 5.7 MB/s \n", "\u001b[K |████████████████████████████████| 58 kB 4.6 MB/s \n", "\u001b[?25h Building wheel for python-multipart (setup.py) ... \u001b[?25l\u001b[?25hdone\n", "\u001b[K |████████████████████████████████| 745 kB 17.9 MB/s \n", "\u001b[?25h Building wheel for pyngrok (setup.py) ... \u001b[?25l\u001b[?25hdone\n", "\u001b[K |████████████████████████████████| 596 kB 31.7 MB/s \n", "\u001b[K |████████████████████████████████| 10.1 MB 43.2 MB/s \n", "\u001b[K |████████████████████████████████| 164 kB 24.5 MB/s \n", "\u001b[K |████████████████████████████████| 181 kB 53.8 MB/s \n", "\u001b[K |████████████████████████████████| 4.3 MB 41.5 MB/s \n", "\u001b[K |████████████████████████████████| 111 kB 71.4 MB/s \n", "\u001b[K |████████████████████████████████| 77 kB 7.4 MB/s \n", "\u001b[K |████████████████████████████████| 63 kB 2.4 MB/s \n", "\u001b[K |████████████████████████████████| 131 kB 77.6 MB/s \n", "\u001b[K |████████████████████████████████| 130 kB 71.4 MB/s \n", "\u001b[K |████████████████████████████████| 793 kB 52.9 MB/s \n", "\u001b[K |████████████████████████████████| 428 kB 64.7 MB/s \n", "\u001b[K |████████████████████████████████| 381 kB 67.1 MB/s \n", "\u001b[?25h Building wheel for blinker (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Building wheel for validators (setup.py) ... \u001b[?25l\u001b[?25hdone\n", "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", "jupyter-console 5.2.0 requires prompt-toolkit<2.0.0,>=1.0.0, but you have prompt-toolkit 3.0.29 which is incompatible.\n", "google-colab 1.0.0 requires ipykernel~=4.10, but you have ipykernel 6.13.0 which is incompatible.\n", "google-colab 1.0.0 requires ipython~=5.5.0, but you have ipython 7.33.0 which is incompatible.\n", "google-colab 1.0.0 requires tornado~=5.1.0; python_version >= \"3.0\", but you have tornado 6.1 which is incompatible.\u001b[0m\n", "\u001b[K |████████████████████████████████| 5.1 MB 35.6 MB/s \n", "\u001b[K |████████████████████████████████| 84 kB 4.2 MB/s \n", "\u001b[K |████████████████████████████████| 54 kB 3.9 MB/s \n", "\u001b[K |████████████████████████████████| 253 kB 67.7 MB/s \n", "\u001b[K |████████████████████████████████| 2.0 MB 64.1 MB/s \n", "\u001b[K |████████████████████████████████| 212 kB 76.2 MB/s \n", "\u001b[K |████████████████████████████████| 11.1 MB 63.1 MB/s \n", "\u001b[K |████████████████████████████████| 63 kB 2.6 MB/s \n", "\u001b[K |████████████████████████████████| 43 kB 2.9 MB/s \n", "\u001b[K |████████████████████████████████| 856 kB 42.9 MB/s \n", "\u001b[K |████████████████████████████████| 62 kB 1.0 MB/s \n", "\u001b[K |████████████████████████████████| 4.0 MB 56.1 MB/s \n", "\u001b[?25h Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n", "Collecting evidently\n", " Downloading evidently-0.1.50.dev0-py3-none-any.whl (11.9 MB)\n", "\u001b[K |████████████████████████████████| 11.9 MB 19.6 MB/s \n", "\u001b[?25hCollecting PyYAML~=5.1\n", " Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n", "\u001b[K |████████████████████████████████| 636 kB 64.2 MB/s \n", "\u001b[?25hCollecting statsmodels>=0.12.2\n", " Downloading statsmodels-0.13.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.8 MB)\n", "\u001b[K |████████████████████████████████| 9.8 MB 59.3 MB/s \n", "\u001b[?25hRequirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.7/dist-packages (from evidently) (1.21.6)\n", "Requirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from evidently) (1.3.5)\n", "Collecting dataclasses>=0.6\n", " Downloading dataclasses-0.6-py3-none-any.whl (14 kB)\n", "Collecting scipy>=1.5.4\n", " Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)\n", "\u001b[K |████████████████████████████████| 38.1 MB 1.2 MB/s \n", "\u001b[?25hRequirement already satisfied: plotly>=5.5.0 in /usr/local/lib/python3.7/dist-packages (from evidently) (5.5.0)\n", "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from evidently) (2.23.0)\n", "Requirement already satisfied: scikit-learn>=0.23.2 in /usr/local/lib/python3.7/dist-packages (from evidently) (1.0.2)\n", "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.1.5->evidently) (2022.1)\n", "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.1.5->evidently) (2.8.2)\n", "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from plotly>=5.5.0->evidently) (1.15.0)\n", "Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from plotly>=5.5.0->evidently) (8.0.1)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->evidently) (1.24.3)\n", "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->evidently) (3.0.4)\n", "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->evidently) (2.10)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->evidently) (2021.10.8)\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.23.2->evidently) (3.1.0)\n", "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.23.2->evidently) (1.1.0)\n", "Requirement already satisfied: patsy>=0.5.2 in /usr/local/lib/python3.7/dist-packages (from statsmodels>=0.12.2->evidently) (0.5.2)\n", "Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.7/dist-packages (from statsmodels>=0.12.2->evidently) (21.3)\n", "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=21.3->statsmodels>=0.12.2->evidently) (3.0.9)\n", "Installing collected packages: scipy, statsmodels, PyYAML, dataclasses, evidently\n", " Attempting uninstall: scipy\n", " Found existing installation: scipy 1.4.1\n", " Uninstalling scipy-1.4.1:\n", " Successfully uninstalled scipy-1.4.1\n", " Attempting uninstall: statsmodels\n", " Found existing installation: statsmodels 0.10.2\n", " Uninstalling statsmodels-0.10.2:\n", " Successfully uninstalled statsmodels-0.10.2\n", " Attempting uninstall: PyYAML\n", " Found existing installation: PyYAML 6.0\n", " Uninstalling PyYAML-6.0:\n", " Successfully uninstalled PyYAML-6.0\n", "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", "albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.\u001b[0m\n", "Successfully installed PyYAML-5.4.1 dataclasses-0.6 evidently-0.1.50.dev0 scipy-1.7.3 statsmodels-0.13.2\n" ] } ] }, { "cell_type": "markdown", "source": [ "Notice that **you may need to restart the kernel** after the above installations." ], "metadata": { "id": "ORfWcYOulopf" } }, { "cell_type": "code", "source": [ "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import requests\n", "import os\n", "import json\n", "import sys\n", "from pyngrok import ngrok, conf\n", "import getpass\n", "import zipfile\n", "import io\n", "\n", "import datetime\n", "\n", "import tensorflow as tf\n", "#tf.compat.v1.disable_v2_behavior() #Should be enabled when using DeepSHAP, see https://github.com/slundberg/shap/issues/2189\n", "from tensorflow import keras\n", "\n", "import bentoml\n", "from sklearn.metrics import accuracy_score, precision_score, confusion_matrix, classification_report\n", "from sklearn.model_selection import train_test_split\n", "from sklearn import svm\n", "from sklearn import ensemble\n", "from sklearn import datasets\n", "\n", "import gradio as gr\n", "\n", "from evidently.dashboard import Dashboard\n", "from evidently.pipeline.column_mapping import ColumnMapping\n", "from evidently.dashboard.tabs import DataDriftTab, NumTargetDriftTab, RegressionPerformanceTab\n", "\n", "import matplotlib as mpl\n", "from matplotlib import pyplot as plt\n", "from matplotlib import cm\n", "%matplotlib inline" ], "metadata": { "id": "5bV_HvPiH-9i", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "d979625a-dede-4d65-eb90-de9e03cd15eb" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.7/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated\n", " \"class\": algorithms.Blowfish,\n", "/usr/local/lib/python3.7/dist-packages/distributed/config.py:20: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n", " defaults = yaml.load(f)\n" ] } ] }, { "cell_type": "markdown", "source": [ "Here are some tips for this notebook https://amitness.com/2020/06/google-colaboratory-tips/ and https://stackoverflow.com/questions/59741453/is-there-a-general-way-to-run-web-applications-on-google-colab." ], "metadata": { "id": "VCxpUaR0Eot3" } }, { "cell_type": "markdown", "source": [ "`ngrok` is a reverse proxy tool that opens secure tunnels from public URLs to localhost, perfect for exposing local web servers, building webhook integrations, enabling SSH access, testing chatbots, demoing from your own machine, and more. In this lab, we will use use https://pyngrok.readthedocs.io/en/latest/integrations.html. However, for production environment, it is recommended to use cloud service such as AWS, GCP or Azure. Please refer to https://pycaret.gitbook.io/docs/get-started/functions/deploy#deploy_model for more details." ], "metadata": { "id": "xyQuXGAscwSG" } }, { "cell_type": "code", "source": [ "print(\"Enter your authtoken, which can be copied from https://dashboard.ngrok.com/auth\")\n", "conf.get_default().auth_token = getpass.getpass()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "mR37ypzgQtw5", "outputId": "791a3805-d01f-46ef-de3b-0f6b96fb242b" }, "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Enter your authtoken, which can be copied from https://dashboard.ngrok.com/auth\n", "··········\n" ] } ] }, { "cell_type": "code", "source": [ "# Setup a tunnel to the streamlit port 8050\n", "public_url = ngrok.connect(8050)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ZX7_8hbYPMyo", "outputId": "a5527021-0a85-42b6-960d-839fb342203b" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "" ] } ] }, { "cell_type": "code", "source": [ "public_url" ], "metadata": { "id": "S7fv6CyzPjMj", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "0c81541f-29e6-40aa-c0d7-dcfa7be8e32c" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " \"http://localhost:8050\">" ] }, "metadata": {}, "execution_count": 5 } ] }, { "cell_type": "markdown", "source": [ "## Deploying TensorFlow models to TensorFlow Serving (TFS)" ], "metadata": { "id": "crG8mceteV3J" } }, { "cell_type": "markdown", "source": [ "You could create your own microservice using any technology you want (e.g., using the Flask library), but why reinvent the wheel when you can just use TF Serving?" ], "metadata": { "id": "cV8Zc1Bqy91q" } }, { "cell_type": "markdown", "source": [ "### Exporting SavedModels" ], "metadata": { "id": "Skjq-RutzUhU" } }, { "cell_type": "markdown", "source": [ "TensorFlow provides a simple `tf.saved_model.save()` function to export models to the SavedModel format. All you need to do is give it the model, specifying its name and version number, and the function will save the model’s computation graph and its weights:" ], "metadata": { "id": "xKWJOHQZzX7I" } }, { "cell_type": "code", "source": [ "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.mnist.load_data()\n", "X_train_full = X_train_full[..., np.newaxis].astype(np.float32) / 255.\n", "X_test = X_test[..., np.newaxis].astype(np.float32) / 255.\n", "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n", "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n", "X_new = X_test[:3]" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Ek0vl3M1yKJy", "outputId": "c3245420-e408-4fe3-ac04-cbd36a2141e0" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n", "11493376/11490434 [==============================] - 0s 0us/step\n", "11501568/11490434 [==============================] - 0s 0us/step\n" ] } ] }, { "cell_type": "code", "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.Flatten(input_shape=[28, 28, 1]),\n", " keras.layers.Dense(100, activation=\"relu\"),\n", " keras.layers.Dense(10, activation=\"softmax\")\n", "])\n", "model.compile(loss=\"sparse_categorical_crossentropy\",\n", " optimizer=keras.optimizers.SGD(learning_rate=1e-2),\n", " metrics=[\"accuracy\"])\n", "model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-V7C_DyLzfOK", "outputId": "cf5a97f4-72b9-4e41-de58-f8f666cd8dc6" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/10\n", "1719/1719 [==============================] - 8s 3ms/step - loss: 0.7012 - accuracy: 0.8241 - val_loss: 0.3715 - val_accuracy: 0.9024\n", "Epoch 2/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3536 - accuracy: 0.9020 - val_loss: 0.2990 - val_accuracy: 0.9144\n", "Epoch 3/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3036 - accuracy: 0.9145 - val_loss: 0.2651 - val_accuracy: 0.9272\n", "Epoch 4/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2736 - accuracy: 0.9231 - val_loss: 0.2436 - val_accuracy: 0.9334\n", "Epoch 5/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2509 - accuracy: 0.9296 - val_loss: 0.2257 - val_accuracy: 0.9364\n", "Epoch 6/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.2322 - accuracy: 0.9350 - val_loss: 0.2121 - val_accuracy: 0.9396\n", "Epoch 7/10\n", "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2161 - accuracy: 0.9400 - val_loss: 0.1970 - val_accuracy: 0.9452\n", "Epoch 8/10\n", "1719/1719 [==============================] - 8s 5ms/step - loss: 0.2021 - accuracy: 0.9432 - val_loss: 0.1880 - val_accuracy: 0.9476\n", "Epoch 9/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1898 - accuracy: 0.9470 - val_loss: 0.1778 - val_accuracy: 0.9524\n", "Epoch 10/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.1793 - accuracy: 0.9493 - val_loss: 0.1685 - val_accuracy: 0.9544\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": {}, "execution_count": 7 } ] }, { "cell_type": "code", "source": [ "np.round(model.predict(X_new), 2)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "w2LJJVPvzwWj", "outputId": "b8be9082-450b-428e-9685-c45406dd945e" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. ],\n", " [0. , 0. , 0.99, 0.01, 0. , 0. , 0. , 0. , 0. , 0. ],\n", " [0. , 0.97, 0.01, 0. , 0. , 0. , 0. , 0.01, 0. , 0. ]],\n", " dtype=float32)" ] }, "metadata": {}, "execution_count": 8 } ] }, { "cell_type": "code", "source": [ "model_version = \"0001\"\n", "model_name = \"my_mnist_model\"\n", "model_path = os.path.join(model_name, model_version)" ], "metadata": { "id": "R7sB3_2rz_KA" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "tf.keras.models.save_model(\n", " model,\n", " model_path,\n", " overwrite=True,\n", " include_optimizer=True,\n", " save_format=None,\n", " signatures=None,\n", " options=None\n", ")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fGL8NoSU3k3w", "outputId": "b7720251-9cf4-4ed3-b64d-d4aae449915f" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "INFO:tensorflow:Assets written to: my_mnist_model/0001/assets\n" ] }, { "output_type": "stream", "name": "stderr", "text": [ "INFO:tensorflow:Assets written to: my_mnist_model/0001/assets\n" ] } ] }, { "cell_type": "markdown", "source": [ "A SavedModel represents a version of your model. It is stored as a directory containing a `saved_model.pb` file, which defines the computation graph (represented as a **serialized protocol buffer**), and a variables subdirectory\n", "containing the variable values. For models containing a large number of weights, these variable values may be split across multiple files. A SavedModel also includes an `assets` subdirectory that may contain additional data, such as vocabulary files, class names, or some example instances for this model." ], "metadata": { "id": "MTwiLc4_13fw" } }, { "cell_type": "code", "source": [ "for root, dirs, files in os.walk(model_name):\n", " indent = ' ' * root.count(os.sep)\n", " print('{}{}/'.format(indent, os.path.basename(root)))\n", " for filename in files:\n", " print('{}{}'.format(indent + ' ', filename))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "j9VjpILs0Hch", "outputId": "c9daf326-b702-4e24-c5f0-be6f67eb8752" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "my_mnist_model/\n", " 0001/\n", " saved_model.pb\n", " keras_metadata.pb\n", " assets/\n", " variables/\n", " variables.index\n", " variables.data-00000-of-00001\n" ] } ] }, { "cell_type": "markdown", "source": [ "As you might expect, you can load a SavedModel using the `tf.keras.models.load_model` function. " ], "metadata": { "id": "qrn8ac6t1aL1" } }, { "cell_type": "code", "source": [ "saved_model = tf.keras.models.load_model(model_path)\n", "y_pred = saved_model(X_new, training=False)" ], "metadata": { "id": "wHPPMDzt0MP4" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "TensorFlow also comes with a small `saved_model_cli` command-line tool to inspect SavedModels:" ], "metadata": { "id": "wlmN7c9X4deT" } }, { "cell_type": "code", "source": [ "!saved_model_cli show --dir {model_path} --all" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "550WGkUX4Mhc", "outputId": "3e53cb22-34f1-47a0-9548-9a9df420d6f6" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n", "\n", "signature_def['__saved_model_init_op']:\n", " The given SavedModel SignatureDef contains the following input(s):\n", " The given SavedModel SignatureDef contains the following output(s):\n", " outputs['__saved_model_init_op'] tensor_info:\n", " dtype: DT_INVALID\n", " shape: unknown_rank\n", " name: NoOp\n", " Method name is: \n", "\n", "signature_def['serving_default']:\n", " The given SavedModel SignatureDef contains the following input(s):\n", " inputs['flatten_input'] tensor_info:\n", " dtype: DT_FLOAT\n", " shape: (-1, 28, 28, 1)\n", " name: serving_default_flatten_input:0\n", " The given SavedModel SignatureDef contains the following output(s):\n", " outputs['dense_1'] tensor_info:\n", " dtype: DT_FLOAT\n", " shape: (-1, 10)\n", " name: StatefulPartitionedCall:0\n", " Method name is: tensorflow/serving/predict\n", "\n", "Concrete Functions:\n", " Function Name: '__call__'\n", " Option #1\n", " Callable with:\n", " Argument #1\n", " inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n", " Argument #2\n", " DType: bool\n", " Value: True\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", " Option #2\n", " Callable with:\n", " Argument #1\n", " flatten_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='flatten_input')\n", " Argument #2\n", " DType: bool\n", " Value: True\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", " Option #3\n", " Callable with:\n", " Argument #1\n", " inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n", " Argument #2\n", " DType: bool\n", " Value: False\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", " Option #4\n", " Callable with:\n", " Argument #1\n", " flatten_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='flatten_input')\n", " Argument #2\n", " DType: bool\n", " Value: False\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", "\n", " Function Name: '_default_save_signature'\n", " Option #1\n", " Callable with:\n", " Argument #1\n", " flatten_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='flatten_input')\n", "\n", " Function Name: 'call_and_return_all_conditional_losses'\n", " Option #1\n", " Callable with:\n", " Argument #1\n", " flatten_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='flatten_input')\n", " Argument #2\n", " DType: bool\n", " Value: True\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", " Option #2\n", " Callable with:\n", " Argument #1\n", " flatten_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='flatten_input')\n", " Argument #2\n", " DType: bool\n", " Value: False\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", " Option #3\n", " Callable with:\n", " Argument #1\n", " inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n", " Argument #2\n", " DType: bool\n", " Value: True\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n", " Option #4\n", " Callable with:\n", " Argument #1\n", " inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='inputs')\n", " Argument #2\n", " DType: bool\n", " Value: False\n", " Argument #3\n", " DType: NoneType\n", " Value: None\n" ] } ] }, { "cell_type": "markdown", "source": [ "A SavedModel contains one or more metagraphs. When you pass a `tf.keras` model, by default the function saves a simple SavedModel: it saves a single metagraph tagged \"serve\", which contains two signature definitions, an initialization function (called `_saved_model_init_op`) and a default serving function (called `serving_default`). When saving a `tf.keras` model, the default serving function corresponds to the model’s `call()` function, which of course makes predictions." ], "metadata": { "id": "7lisp5Ox5KPZ" } }, { "cell_type": "markdown", "source": [ "### Serve your model with TensorFlow Serving" ], "metadata": { "id": "y8lzCevs56av" } }, { "cell_type": "markdown", "source": [ "We're preparing to install TensorFlow Serving using [Aptitude](https://wiki.debian.org/Aptitude) since this Colab runs in a Debian environment. We'll add the `tensorflow-model-server` package to the list of packages that Aptitude knows about. Note that we're running as root." ], "metadata": { "id": "vVhUpbsP6io2" } }, { "cell_type": "code", "source": [ "# We need sudo prefix if not on a Google Colab.\n", "if 'google.colab' not in sys.modules:\n", " SUDO_IF_NEEDED = 'sudo'\n", "else:\n", " SUDO_IF_NEEDED = ''" ], "metadata": { "id": "xANYl8LW4mhv" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "# This is the same as you would do from your command line, but without the [arch=amd64], and no sudo\n", "# You would instead do:\n", "# echo \"deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \\\n", "# curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -\n", "\n", "!echo \"deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\" | {SUDO_IF_NEEDED} tee /etc/apt/sources.list.d/tensorflow-serving.list && \\\n", "curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | {SUDO_IF_NEEDED} apt-key add -\n", "!{SUDO_IF_NEEDED} apt update" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4Kn7Pa7p6aWC", "outputId": "27ffb914-1385-4b96-b374-c69e38ec4946" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\n", " % Total % Received % Xferd Average Speed Time Time Time Current\n", " Dload Upload Total Spent Left Speed\n", "100 2943 100 2943 0 0 17110 0 --:--:-- --:--:-- --:--:-- 17110\n", "OK\n", "Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease [1,581 B]\n", "Get:2 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]\n", "Ign:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease\n", "Hit:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release\n", "Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease\n", "Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n", "Get:7 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]\n", "Get:8 http://storage.googleapis.com/tensorflow-serving-apt stable InRelease [3,012 B]\n", "Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages [748 kB]\n", "Get:10 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n", "Get:11 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ Packages [85.2 kB]\n", "Hit:12 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n", "Get:14 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n", "Get:15 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease [15.9 kB]\n", "Get:16 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]\n", "Get:17 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages [341 B]\n", "Get:18 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server-universal amd64 Packages [349 B]\n", "Get:19 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,957 kB]\n", "Get:20 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [932 kB]\n", "Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,277 kB]\n", "Get:22 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,503 kB]\n", "Get:23 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,761 kB]\n", "Get:24 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [1,004 kB]\n", "Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [966 kB]\n", "Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,195 kB]\n", "Get:27 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 Packages [45.3 kB]\n", "Get:28 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [44.3 kB]\n", "Fetched 15.8 MB in 3s (4,746 kB/s)\n", "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", "66 packages can be upgraded. Run 'apt list --upgradable' to see them.\n" ] } ] }, { "cell_type": "code", "source": [ "!{SUDO_IF_NEEDED} apt-get install tensorflow-model-server" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5Zqq_CaL6smd", "outputId": "64a1ac1a-5865-4993-914b-85888cf79333" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", "The following packages were automatically installed and are no longer required:\n", " libnvidia-common-460 nsight-compute-2020.2.0\n", "Use 'apt autoremove' to remove them.\n", "The following NEW packages will be installed:\n", " tensorflow-model-server\n", "0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded.\n", "Need to get 340 MB of archives.\n", "After this operation, 0 B of additional disk space will be used.\n", "Get:1 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 tensorflow-model-server all 2.8.0 [340 MB]\n", "Fetched 340 MB in 4s (80.1 MB/s)\n", "Selecting previously unselected package tensorflow-model-server.\n", "(Reading database ... 155203 files and directories currently installed.)\n", "Preparing to unpack .../tensorflow-model-server_2.8.0_all.deb ...\n", "Unpacking tensorflow-model-server (2.8.0) ...\n", "Setting up tensorflow-model-server (2.8.0) ...\n" ] } ] }, { "cell_type": "markdown", "source": [ "This is where we start running TensorFlow Serving and load our model. After it loads we can start making inference requests using REST. There are some important parameters:\n", "\n", "* `rest_api_port`: The port that you'll use for REST requests.\n", "* `model_name`: You'll use this in the URL of REST requests. It can be anything.\n", "* `model_base_path`: This is the path to the directory where you've saved your model." ], "metadata": { "id": "lbp5_Vf39fFu" } }, { "cell_type": "code", "source": [ "os.environ[\"MODEL_DIR\"] = os.path.split(os.path.abspath(model_path))[0]" ], "metadata": { "id": "smENzAY76z0V" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "%%bash --bg\n", "nohup tensorflow_model_server \\\n", " --rest_api_port=8050 \\\n", " --model_name=my_mnist_model \\\n", " --model_base_path=\"${MODEL_DIR}\" > server.log 2>&1" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "PBGHxWHr7C7n", "outputId": "9ff8c11a-87c3-4a39-9c34-b1b6a70105e8" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Starting job # 0 in a separate thread.\n" ] } ] }, { "cell_type": "code", "source": [ "!tail server.log" ], "metadata": { "id": "DAuShYuE7H2P" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "### Querying TF Serving through the REST API" ], "metadata": { "id": "75N3zNzk92fg" } }, { "cell_type": "markdown", "source": [ "Let’s start by creating the query. It must contain the name of the function signature you want to call, and of course the input data:" ], "metadata": { "id": "xD8fQFTv-D2e" } }, { "cell_type": "code", "source": [ "input_data_json = json.dumps({\n", " \"signature_name\": \"serving_default\",\n", " \"instances\": X_new.tolist(),\n", "})" ], "metadata": { "id": "tx5imAfy7W8x" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Note that the JSON format is 100% text-based, so the `X_new` NumPy array had to be converted to a Python list and then formatted as JSON:" ], "metadata": { "id": "gJ9fpeDQ-MTo" } }, { "cell_type": "code", "source": [ "repr(input_data_json)[:1500] + \"...\"" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 122 }, "id": "tzPnf4Tk7pn0", "outputId": "0f0c9353-7d29-4752-e5e4-68cf0806a7aa" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'\\'{\"signature_name\": \"serving_default\", \"instances\": [[[[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.32941177487373...'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 21 } ] }, { "cell_type": "markdown", "source": [ "Now let’s send the input data to TF Serving by sending an HTTP POST request. This can be done easily using the requests library:" ], "metadata": { "id": "uvg-AMbO-T4g" } }, { "cell_type": "code", "source": [ "SERVER_URL = 'http://localhost:8050/v1/models/my_mnist_model:predict'\n", "response = requests.post(SERVER_URL, data=input_data_json)\n", "response.raise_for_status() # raise an exception in case of error\n", "response = response.json()" ], "metadata": { "id": "KgBfYn8e7rNz" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "The response is a dictionary containing a single \"predictions\" key. The corresponding value is the list of predictions. This list is a Python list, so let’s convert it to a NumPy array and round the floats it contains to the\n", "second decimal:" ], "metadata": { "id": "fFyZFS9V-cA4" } }, { "cell_type": "code", "source": [ "y_proba = np.array(response[\"predictions\"])\n", "y_proba.round(2)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-K8iZuQo7xn1", "outputId": "28c6f0b0-a436-4aa7-835e-66e21e255445" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. ],\n", " [0. , 0. , 0.99, 0.01, 0. , 0. , 0. , 0. , 0. , 0. ],\n", " [0. , 0.97, 0.01, 0. , 0. , 0. , 0. , 0.01, 0. , 0. ]])" ] }, "metadata": {}, "execution_count": 25 } ] }, { "cell_type": "markdown", "source": [ "For more information, please refer to https://github.com/tensorflow/serving which include the usuage of gRPC." ], "metadata": { "id": "b6f4iUDeAegQ" } }, { "cell_type": "markdown", "source": [ "### Deploying a new model version" ], "metadata": { "id": "8UGzlBoKA6OA" } }, { "cell_type": "markdown", "source": [ "Now let’s create a new model version and export a SavedModel to the `my_mnist_model/0002` directory, just like earlier:" ], "metadata": { "id": "XoatJ_43BTjY" } }, { "cell_type": "code", "source": [ "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "model = keras.models.Sequential([\n", " keras.layers.Flatten(input_shape=[28, 28, 1]),\n", " keras.layers.Dense(50, activation=\"relu\"),\n", " keras.layers.Dense(50, activation=\"relu\"),\n", " keras.layers.Dense(10, activation=\"softmax\")\n", "])\n", "model.compile(loss=\"sparse_categorical_crossentropy\",\n", " optimizer=keras.optimizers.SGD(learning_rate=1e-2),\n", " metrics=[\"accuracy\"])\n", "history = model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "b4gE94r57zBE", "outputId": "ade61c8d-4a28-4540-c0b5-812244cd0483" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Epoch 1/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7039 - accuracy: 0.8056 - val_loss: 0.3418 - val_accuracy: 0.9042\n", "Epoch 2/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3204 - accuracy: 0.9082 - val_loss: 0.2674 - val_accuracy: 0.9242\n", "Epoch 3/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2650 - accuracy: 0.9235 - val_loss: 0.2227 - val_accuracy: 0.9368\n", "Epoch 4/10\n", "1719/1719 [==============================] - 7s 4ms/step - loss: 0.2319 - accuracy: 0.9329 - val_loss: 0.2032 - val_accuracy: 0.9432\n", "Epoch 5/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2089 - accuracy: 0.9399 - val_loss: 0.1833 - val_accuracy: 0.9482\n", "Epoch 6/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.1908 - accuracy: 0.9446 - val_loss: 0.1740 - val_accuracy: 0.9498\n", "Epoch 7/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1756 - accuracy: 0.9490 - val_loss: 0.1605 - val_accuracy: 0.9540\n", "Epoch 8/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1631 - accuracy: 0.9524 - val_loss: 0.1543 - val_accuracy: 0.9558\n", "Epoch 9/10\n", "1719/1719 [==============================] - 4s 3ms/step - loss: 0.1517 - accuracy: 0.9568 - val_loss: 0.1459 - val_accuracy: 0.9574\n", "Epoch 10/10\n", "1719/1719 [==============================] - 5s 3ms/step - loss: 0.1429 - accuracy: 0.9583 - val_loss: 0.1358 - val_accuracy: 0.9616\n" ] } ] }, { "cell_type": "code", "source": [ "model_version = \"0002\"\n", "model_name = \"my_mnist_model\"\n", "model_path = os.path.join(model_name, model_version)\n", "\n", "tf.keras.models.save_model(\n", " model,\n", " model_path,\n", " overwrite=True,\n", " include_optimizer=True,\n", " save_format=None,\n", " signatures=None,\n", " options=None\n", ")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "rCx3wneqA-ld", "outputId": "1e90f026-e4af-4bd0-da21-8671cc0f9e93" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "INFO:tensorflow:Assets written to: my_mnist_model/0002/assets\n" ] }, { "output_type": "stream", "name": "stderr", "text": [ "INFO:tensorflow:Assets written to: my_mnist_model/0002/assets\n" ] } ] }, { "cell_type": "code", "source": [ "for root, dirs, files in os.walk(model_name):\n", " indent = ' ' * root.count(os.sep)\n", " print('{}{}/'.format(indent, os.path.basename(root)))\n", " for filename in files:\n", " print('{}{}'.format(indent + ' ', filename))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Kz6jGDHhBJHQ", "outputId": "856e6eaa-7fbd-4cee-9415-08e3736262f6" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "my_mnist_model/\n", " 0001/\n", " saved_model.pb\n", " keras_metadata.pb\n", " assets/\n", " variables/\n", " variables.index\n", " variables.data-00000-of-00001\n", " 0002/\n", " saved_model.pb\n", " keras_metadata.pb\n", " assets/\n", " variables/\n", " variables.index\n", " variables.data-00000-of-00001\n" ] } ] }, { "cell_type": "markdown", "source": [ "At regular intervals (the delay is configurable), TensorFlow Serving checks for new model versions. If it finds one, it will automatically handle the transition gracefully: by default, it will answer pending requests (if any) with the previous model version, while handling new requests with the new version" ], "metadata": { "id": "JNiIxJDABhvx" } }, { "cell_type": "code", "source": [ "SERVER_URL = 'http://localhost:8050/v1/models/my_mnist_model:predict'\n", " \n", "response = requests.post(SERVER_URL, data=input_data_json)\n", "response.raise_for_status()\n", "response = response.json()" ], "metadata": { "id": "bykQlJsvBLir" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "y_proba = np.array(response[\"predictions\"])\n", "y_proba.round(2)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4FThGiclBOHo", "outputId": "bce6b6de-f41e-4cea-f5a8-0dfd408c9f37" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. ],\n", " [0. , 0. , 0.99, 0.01, 0. , 0. , 0. , 0. , 0. , 0. ],\n", " [0. , 0.99, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]])" ] }, "metadata": {}, "execution_count": 34 } ] }, { "cell_type": "code", "source": [ "!pgrep tensorflow" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AspWanW5KfZP", "outputId": "1b684ebd-8106-49fd-da2e-9655fb98ef21" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "1599\n" ] } ] }, { "cell_type": "code", "source": [ "!kill 1599" ], "metadata": { "id": "VZonNdWhKhdX" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "As you can see, TF Serving makes it quite simple to deploy new models. Moreover, if you discover that version 2 does not work as well as you expected, then rolling back to version 1 is as simple as removing the `my_mnist_model/0002` directory." ], "metadata": { "id": "L6eqGpg-Bwme" } }, { "cell_type": "markdown", "source": [ "You can also refer to https://github.com/microsoft/ML-For-Beginners/blob/main/3-Web-App/1-Web-App/README.md or https://github.com/rodrigo-arenas/fast-ml-deploy which use [Flask](https://flask.palletsprojects.com/en/2.1.x/) and [FastAPI](https://fastapi.tiangolo.com/) that may have more flexibility. " ], "metadata": { "id": "UgV76GJjDJiZ" } }, { "cell_type": "markdown", "source": [ "## Deploy a REST API server using BentoML" ], "metadata": { "id": "5HnKXdgmhhTv" } }, { "cell_type": "markdown", "source": [ "### Train a classifier model using the iris dataset" ], "metadata": { "id": "xghZJBNqhs0O" } }, { "cell_type": "code", "source": [ "# Load training data\n", "iris = datasets.load_iris()\n", "X, y = iris.data, iris.target\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)\n", "\n", "# Model Training\n", "clf = svm.SVC()\n", "clf.fit(X_train, y_train)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "N6RLmbgHho9o", "outputId": "c2ba8b88-6c0e-4098-9571-5b52925ddfd6" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "SVC()" ] }, "metadata": {}, "execution_count": 37 } ] }, { "cell_type": "code", "source": [ "y_pred = clf.predict(X_test)\n", "print(classification_report(y_test,y_pred))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "TC0bmn6ihxRY", "outputId": "9c9b09e2-cd11-4075-c2a5-1952011e9c84" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " precision recall f1-score support\n", "\n", " 0 1.00 1.00 1.00 10\n", " 1 1.00 1.00 1.00 9\n", " 2 1.00 1.00 1.00 11\n", "\n", " accuracy 1.00 30\n", " macro avg 1.00 1.00 1.00 30\n", "weighted avg 1.00 1.00 1.00 30\n", "\n" ] } ] }, { "cell_type": "markdown", "source": [ "Save the `clf` model with BentoML. We begin by saving a trained model instance to **BentoML’s local model store**. The local model store is used to version your models as well as control which models are packaged with your bento.\n", "\n", "If the models you wish to use are already saved to disk or available in a cloud repository, they can also be added to BentoML with the [import APIs](https://docs.bentoml.org/en/latest/concepts/bento_management.html#bento-management-page). It is noted that there are a [wide range of models](https://docs.bentoml.org/en/latest/frameworks/index.html#frameworks-page) can be saved via bentoml." ], "metadata": { "id": "Q8zYLNH_itMe" } }, { "cell_type": "code", "source": [ "try:\n", " bentoml.sklearn.save(\"iris_clf\", clf) # Use YAML for dump\n", "except:\n", " print(\"Have some exception, but the model still save sucessfully\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "8Tr-eMEYimGA", "outputId": "b0860373-4480-42cf-995c-d26c4f0e94aa" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "ERROR:opentelemetry.context:Failed to load context: contextvars_context\n" ] }, { "output_type": "stream", "name": "stdout", "text": [ "Have some exception, but the model still save sucessfully\n" ] } ] }, { "cell_type": "markdown", "source": [ "Models saved can be accessed via `bentoml models` CLI command:" ], "metadata": { "id": "GdbaGBQGjkT3" } }, { "cell_type": "code", "source": [ "!bentoml models list" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AMj0y2KRiwkZ", "outputId": "5c508579-a030-452c-9c38-7370178bb640" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\u001b[1m \u001b[0m\u001b[1mTag \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mModule \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mSize \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mCreation Time \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mPath \u001b[0m\u001b[1m \u001b[0m\n", " iris_clf:rqrik2… bentoml.sklearn 5.38 KiB 2022-05-15 ~/bentoml/model… \n", " 07:49:28 \n" ] } ] }, { "cell_type": "markdown", "source": [ "Test loading the model as a BentoML Runner instance:" ], "metadata": { "id": "ABLTgSPMjuOG" } }, { "cell_type": "code", "source": [ "test_runner = bentoml.sklearn.load_runner(\"iris_clf:latest\")\n", "test_runner.run([5.9, 3. , 5.1, 1.8]) # => array(2)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "njBF04-Yjfek", "outputId": "725bf9e4-8b17-4892-ea98-ceb3dbd1bcef" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array(2)" ] }, "metadata": {}, "execution_count": 41 } ] }, { "cell_type": "markdown", "source": [ "### Create a BentoML Service for serving the model" ], "metadata": { "id": "UsAvuehvj1zR" } }, { "cell_type": "markdown", "source": [ "Services are the core components of BentoML, where the serving logic is defined. With the model saved in the model store, we can define the [`service`](https://docs.bentoml.org/en/latest/concepts/service_definition.html#service-definition-page) by creating a Python file service.py with the following contents:" ], "metadata": { "id": "g2wgm8sFk5oY" } }, { "cell_type": "code", "source": [ "%%writefile service.py\n", "import numpy as np\n", "import bentoml\n", "from bentoml.io import NumpyNdarray\n", "\n", "# Load the runner for the latest ScikitLearn model we just saved\n", "iris_clf_runner = bentoml.sklearn.load_runner(\"iris_clf:latest\")\n", "\n", "# Create the iris_classifier service with the ScikitLearn runner\n", "# Multiple runners may be specified if needed in the runners array\n", "# When packaged as a bento, the runners here will included\n", "svc = bentoml.Service(\"iris_classifier\", runners=[iris_clf_runner])\n", "\n", "# Create API function with pre- and post- processing logic with your new \"svc\" annotation\n", "@svc.api(input=NumpyNdarray(), output=NumpyNdarray())\n", "def classify(input_series: np.ndarray) -> np.ndarray:\n", " # Define pre-processing logic\n", " result = iris_clf_runner.run(input_series)\n", " # Define post-processing logic\n", " return result" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UlhoGXS8jxOA", "outputId": "bcbf5a30-5c76-4f9d-f707-9af3b04c8adc" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing service.py\n" ] } ] }, { "cell_type": "markdown", "source": [ "In this example, we defined the input and output type to be `numpy.ndarray`. More options, such as `pandas.DataFrame` and `PIL.image` are also supported. " ], "metadata": { "id": "xz5SAEJulUG_" } }, { "cell_type": "markdown", "source": [ "We now have everything we need to serve our first request. Launch the server in debug mode by running the `bentoml serve` command in the current working directory. Using the `--reload` option allows the server to reflect any changes made to the `service.py` module without restarting:" ], "metadata": { "id": "i_ZCPRN9lsLY" } }, { "cell_type": "code", "source": [ "!nohup bentoml serve ./service.py:svc --reload --port 8050 &" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6oLpk4yqoZFH", "outputId": "cd84bc47-1eb8-4dcb-e023-c3a0e4568705" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "nohup: appending output to 'nohup.out'\n" ] } ] }, { "cell_type": "markdown", "source": [ "We can then send requests to the newly started service with any HTTP client:" ], "metadata": { "id": "IbSWqAIPuPGf" } }, { "cell_type": "code", "source": [ "requests.post(\n", " \"http://127.0.0.1:8050/classify\",\n", " headers={\"content-type\": \"application/json\"},\n", " data=\"[5.9,3,5.1,1.8]\").text" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "id": "F-cNwCoXl6u3", "outputId": "263b687f-4842-4d29-8f74-59329a7af40b" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'2'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 44 } ] }, { "cell_type": "code", "source": [ "!pgrep bentoml" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_T7XLWIzmPlg", "outputId": "846c11c3-1adb-4571-cafd-83e5e71fcc0e" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "1731\n" ] } ] }, { "cell_type": "code", "source": [ "!kill 1731" ], "metadata": { "id": "z5jPb8BrqSMT" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "### Build and Deploy Bentos" ], "metadata": { "id": "ldMJBh23vEc_" } }, { "cell_type": "markdown", "source": [ "Once we are happy with the service definition, we can build the model and service into a bento. Bentos are the distribution format for services, and contains all the information required to run or deploy those services, such as models and dependencies. For more information about building bentos, see [Building Bentos](https://docs.bentoml.org/en/latest/guides/building_bentos.html#building-bentos-page)." ], "metadata": { "id": "TZ4o7ucOvF4-" } }, { "cell_type": "markdown", "source": [ "To build a Bento, first create a file named `bentofile.yaml` in your project directory:" ], "metadata": { "id": "L5qUMltdvcBu" } }, { "cell_type": "code", "source": [ "%%writefile bentofile.yaml\n", "service: \"service.py:svc\" # A convention for locating your service: :\n", "description: \"file: ./README.md\"\n", "labels:\n", " owner: bentoml-team\n", " stage: demo\n", "include:\n", " - \"*.py\" # A pattern for matching which files to include in the bento\n", "python:\n", " packages:\n", " - scikit-learn # Additional libraries to be included in the bento\n", " - pandas" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "l0sEOEowu4w0", "outputId": "b496b52d-5633-41af-c54c-1811b294922a" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing bentofile.yaml\n" ] } ] }, { "cell_type": "code", "source": [ "%%writefile README.md\n", "This is a iris classifier" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Vj5BPdZPv06u", "outputId": "1869ad3e-1abe-4c19-c3af-f59dcc88256a" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing README.md\n" ] } ] }, { "cell_type": "markdown", "source": [ "Next, use the bentoml build CLI command in the same directory to build a bento." ], "metadata": { "id": "oq-lbzYhvgI3" } }, { "cell_type": "code", "source": [ "!bentoml build" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "eAA9xJnAvYa_", "outputId": "8882f155-d021-44dc-e85f-af3737e0b18f" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\u001b[2;36m05/15/2022 07:50:40 AM\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m \u001b[1m[\u001b[0mcli\u001b[1m]\u001b[0m Building BentoML service \n", "\u001b[2;36m \u001b[0m \u001b[32m\"iris_classifier:w5ck3iwueoi64asc\"\u001b[0m from build \n", "\u001b[2;36m \u001b[0m context \u001b[32m\"/content\"\u001b[0m \n", "\u001b[2;36m05/15/2022 07:50:40 AM\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m \u001b[1m[\u001b[0mcli\u001b[1m]\u001b[0m Packing model \u001b[32m\"iris_clf:rqrik2gueo4bqasc\"\u001b[0m \n", "\u001b[2;36m \u001b[0m from \n", "\u001b[2;36m \u001b[0m \u001b[32m\"/root/bentoml/models/iris_clf/rqrik2gueo4bqasc\"\u001b[0m\n", "\u001b[2;36m05/15/2022 07:50:40 AM\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m \u001b[1m[\u001b[0mcli\u001b[1m]\u001b[0m Successfully saved \n", "\u001b[2;36m \u001b[0m \u001b[1;35mModel\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtag\u001b[0m=\u001b[32m\"iris_clf\u001b[0m\u001b[32m:rqrik2gueo4bqasc\"\u001b[0m, \u001b[33mpath\u001b[0m=\u001b[32m\"/tm\u001b[0m\n", "\u001b[2;36m \u001b[0m \u001b[32mp/tmpy3adr2h6bentoml_bento_iris_classifier/model\u001b[0m\n", "\u001b[2;36m \u001b[0m \u001b[32ms/iris_clf/rqrik2gueo4bqasc/\"\u001b[0m\u001b[1m)\u001b[0m \n", "\u001b[2;36m05/15/2022 07:50:40 AM\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m \u001b[1m[\u001b[0mcli\u001b[1m]\u001b[0m Locking PyPI package versions.. \n", "\u001b[K |████████████████████████████████| 24.8 MB 1.3 MB/s \n", "\u001b[K |████████████████████████████████| 11.3 MB 48.9 MB/s \n", "\u001b[K |████████████████████████████████| 38.1 MB 275 kB/s \n", "\u001b[K |████████████████████████████████| 15.7 MB 39.0 MB/s \n", "\u001b[K |████████████████████████████████| 503 kB 58.9 MB/s \n", "\u001b[K |████████████████████████████████| 306 kB 66.3 MB/s \n", "\u001b[K |████████████████████████████████| 247 kB 69.0 MB/s \n", "\u001b[2;36m05/15/2022 07:50:49 AM\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m \u001b[1m[\u001b[0mcli\u001b[1m]\u001b[0m \n", "\u001b[2;36m \u001b[0m ██████╗░███████╗███╗░░██╗████████╗░█████╗░███╗░░\n", "\u001b[2;36m \u001b[0m ░███╗██╗░░░░░ \n", "\u001b[2;36m \u001b[0m ██╔══██╗██╔════╝████╗░██║╚══██╔══╝██╔══██╗████╗░\n", "\u001b[2;36m \u001b[0m ████║██║░░░░░ \n", "\u001b[2;36m \u001b[0m ██████╦╝█████╗░░██╔██╗██║░░░██║░░░██║░░██║██╔███\n", "\u001b[2;36m \u001b[0m █╔██║██║░░░░░ \n", "\u001b[2;36m \u001b[0m ██╔══██╗██╔══╝░░██║╚████║░░░██║░░░██║░░██║██║╚██\n", "\u001b[2;36m \u001b[0m ╔╝██║██║░░░░░ \n", "\u001b[2;36m \u001b[0m ██████╦╝███████╗██║░╚███║░░░██║░░░╚█████╔╝██║░╚═\n", "\u001b[2;36m \u001b[0m ╝░██║███████╗ \n", "\u001b[2;36m \u001b[0m ╚═════╝░╚══════╝╚═╝░░╚══╝░░░╚═╝░░░░╚════╝░╚═╝░░░\n", "\u001b[2;36m \u001b[0m ░░╚═╝╚══════╝ \n", "\u001b[2;36m \u001b[0m \n", "\u001b[2;36m05/15/2022 07:50:49 AM\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m \u001b[1m[\u001b[0mcli\u001b[1m]\u001b[0m Successfully built \n", "\u001b[2;36m \u001b[0m \u001b[1;35mBento\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtag\u001b[0m=\u001b[32m\"iris_classifier\u001b[0m\u001b[32m:w5ck3iwueoi64asc\"\u001b[0m\u001b[1m)\u001b[0m at\n", "\u001b[2;36m \u001b[0m \u001b[32m\"/root/bentoml/bentos/iris_classifier/w5ck3iwueo\u001b[0m\n", "\u001b[2;36m \u001b[0m \u001b[32mi64asc/\"\u001b[0m \n", "\u001b[?25h" ] } ] }, { "cell_type": "markdown", "source": [ "Bentos built will be saved in the local bento store, which you can view using the bentoml list CLI command." ], "metadata": { "id": "4zwY6y2xwDdf" } }, { "cell_type": "code", "source": [ "!bentoml list" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yIlQ2J8qvnD5", "outputId": "b1aa3de5-d641-4fce-81c3-62e4e28e072c" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\u001b[1m \u001b[0m\u001b[1mTag \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mSize \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mCreation Time \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mPath \u001b[0m\u001b[1m \u001b[0m\n", " iris_classifier:w5ck3… 14.45 KiB 2022-05-15 07:50:49 ~/bentoml/bentos/iris… \n" ] } ] }, { "cell_type": "markdown", "source": [ "We can serve bentos from the bento store using the bentoml serve `--production` CLI command. Using the `--production` option will serve the bento in production mode." ], "metadata": { "id": "lTmUPXn7wPFu" } }, { "cell_type": "code", "source": [ "%%bash --bg\n", "nohup bentoml serve iris_classifier:latest \\\n", " --production \\\n", " --port 8050 > bentoml.log 2>&1" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_IlkMx2Is1ap", "outputId": "1a94b6ab-159d-44bb-be08-a2dd70ebe2ad" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Starting job # 2 in a separate thread.\n" ] } ] }, { "cell_type": "markdown", "source": [ "This is another way to query the server" ], "metadata": { "id": "fv4Ai48uwu9u" } }, { "cell_type": "code", "source": [ "!curl \\\n", " -X POST \\\n", " -H \"content-type: application/json\" \\\n", " --data \"[5.9,3,5.1,1.8]\" \\\n", " http://127.0.0.1:8050/classify" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "KgAcBHm3wbDI", "outputId": "6c7812f5-fba5-4ee1-89c7-0c30c846b5b3" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "2" ] } ] }, { "cell_type": "markdown", "source": [ "The Bento directory contains all code, files, models and configs required for running this service. BentoML standarlizes this file structure which enables serving runtimes and deployment tools to be built on top of it. By default, Bentos are managed under the ~/bentoml/bentos directory:" ], "metadata": { "id": "PbXF0hyfw_og" } }, { "cell_type": "code", "source": [ "path =\"/root/bentoml/bentos/iris_classifier/\"" ], "metadata": { "id": "3agYCiqHwqhI" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "for root, dirs, files in os.walk(path):\n", " indent = ' ' * root.count(os.sep)\n", " print('{}{}/'.format(indent, os.path.basename(root)))\n", " for filename in files:\n", " print('{}{}'.format(indent + ' ', filename))" ], "metadata": { "id": "tUKsg5nM0OsH", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "030b356d-08b7-4212-f089-952f9a4a0558" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " /\n", " latest\n", " w5ck3iwueoi64asc/\n", " bento.yaml\n", " README.md\n", " env/\n", " docker/\n", " Dockerfile\n", " init.sh\n", " entrypoint.sh\n", " python/\n", " requirements.lock.txt\n", " version.txt\n", " requirements.txt\n", " conda/\n", " models/\n", " iris_clf/\n", " latest\n", " rqrik2gueo4bqasc/\n", " model.yaml\n", " saved_model.pkl\n", " src/\n", " service.py\n", " __pycache__/\n", " service.cpython-37.pyc\n", " apis/\n", " openapi.yaml\n" ] } ] }, { "cell_type": "code", "source": [ "!pgrep bentoml" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ke_qrRHQzj0n", "outputId": "97d0bcdd-e5da-4cf2-9c7f-5dfcf5c48db8" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "1815\n" ] } ] }, { "cell_type": "code", "source": [ "!kill 1815" ], "metadata": { "id": "hNslyi-QzkxH" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "For more information, please refer to https://docs.bentoml.org/en/latest/index.html" ], "metadata": { "id": "Bw_A1VnIyReH" } }, { "cell_type": "markdown", "source": [ "## Deploy web base application using streamit" ], "metadata": { "id": "SB48ta6AGzYI" } }, { "cell_type": "markdown", "source": [ "Streamlit's simple and focused API lets you build incredibly rich and powerful tools. It contains a large number of [elements](https://docs.streamlit.io/library/api-reference) and [components](https://streamlit.io/components) that you can use." ], "metadata": { "id": "3BN-1OnBuWpa" } }, { "cell_type": "code", "source": [ "!wget https://raw.githubusercontent.com/dataprofessor/code/master/streamlit/part2/iris-ml-app.py" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "eeYFfVQNBzN9", "outputId": "72077410-eca0-4c26-92ce-8449f4084ee3" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "--2022-05-15 07:52:43-- https://raw.githubusercontent.com/dataprofessor/code/master/streamlit/part2/iris-ml-app.py\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1350 (1.3K) [text/plain]\n", "Saving to: ‘iris-ml-app.py’\n", "\n", "\riris-ml-app.py 0%[ ] 0 --.-KB/s \riris-ml-app.py 100%[===================>] 1.32K --.-KB/s in 0s \n", "\n", "2022-05-15 07:52:43 (66.3 MB/s) - ‘iris-ml-app.py’ saved [1350/1350]\n", "\n" ] } ] }, { "cell_type": "code", "source": [ "!cat iris-ml-app.py" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "aIPcnBo2tKSp", "outputId": "cee15f67-0830-4449-fdd6-29c7549c3ac5" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "import streamlit as st\r\n", "import pandas as pd\r\n", "from sklearn import datasets\r\n", "from sklearn.ensemble import RandomForestClassifier\r\n", "\r\n", "st.write(\"\"\"\r\n", "# Simple Iris Flower Prediction App\r\n", "\r\n", "This app predicts the **Iris flower** type!\r\n", "\"\"\")\r\n", "\r\n", "st.sidebar.header('User Input Parameters')\r\n", "\r\n", "def user_input_features():\r\n", " sepal_length = st.sidebar.slider('Sepal length', 4.3, 7.9, 5.4)\r\n", " sepal_width = st.sidebar.slider('Sepal width', 2.0, 4.4, 3.4)\r\n", " petal_length = st.sidebar.slider('Petal length', 1.0, 6.9, 1.3)\r\n", " petal_width = st.sidebar.slider('Petal width', 0.1, 2.5, 0.2)\r\n", " data = {'sepal_length': sepal_length,\r\n", " 'sepal_width': sepal_width,\r\n", " 'petal_length': petal_length,\r\n", " 'petal_width': petal_width}\r\n", " features = pd.DataFrame(data, index=[0])\r\n", " return features\r\n", "\r\n", "df = user_input_features()\r\n", "\r\n", "st.subheader('User Input parameters')\r\n", "st.write(df)\r\n", "\r\n", "iris = datasets.load_iris()\r\n", "X = iris.data\r\n", "Y = iris.target\r\n", "\r\n", "clf = RandomForestClassifier()\r\n", "clf.fit(X, Y)\r\n", "\r\n", "prediction = clf.predict(df)\r\n", "prediction_proba = clf.predict_proba(df)\r\n", "\r\n", "st.subheader('Class labels and their corresponding index number')\r\n", "st.write(iris.target_names)\r\n", "\r\n", "st.subheader('Prediction')\r\n", "st.write(iris.target_names[prediction])\r\n", "#st.write(prediction)\r\n", "\r\n", "st.subheader('Prediction Probability')\r\n", "st.write(prediction_proba)\r\n" ] } ] }, { "cell_type": "code", "source": [ "%%bash --bg \n", "streamlit run iris-ml-app.py --server.port 8050 > debug.log 2>&1" ], "metadata": { "id": "gLBRsyy0G6Uo", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "cc2126b0-1ee5-4d9e-9be3-c89b91839246" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Starting job # 3 in a separate thread.\n" ] } ] }, { "cell_type": "code", "source": [ "!tail debug.log" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "FFZL2c_5Kaso", "outputId": "5bc734f5-5893-4b91-8a9b-dffff92c8102" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "2022-05-15 07:54:05.807 INFO numexpr.utils: NumExpr defaulting to 2 threads.\n", "\n", " You can now view your Streamlit app in your browser.\n", "\n", " Network URL: http://172.28.0.2:8050\n", " External URL: http://35.243.184.210:8050\n", "\n" ] } ] }, { "cell_type": "code", "source": [ "public_url" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LJuBFXjdtkJA", "outputId": "14c4e8a0-9011-414a-b063-01db4746b914" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " \"http://localhost:8050\">" ] }, "metadata": {}, "execution_count": 64 } ] }, { "cell_type": "markdown", "source": [ "Try to click the above link to access the web app. For more information, please refer to https://github.com/streamlit/streamlit" ], "metadata": { "id": "-JN1a3SutjUI" } }, { "cell_type": "code", "source": [ "!pgrep streamlit" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DhJbvI0_t-Hx", "outputId": "ce28ef46-3918-4252-c7cb-71a7bfbd84a5" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "1866\n" ] } ] }, { "cell_type": "code", "source": [ "!kill 1866" ], "metadata": { "id": "R-6Fhs4auDgR" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Deploy web base application using Gradio" ], "metadata": { "id": "mJsFDLpWQ7HK" } }, { "cell_type": "markdown", "source": [ "UI models are perfect to use with Gradio's image input component, so in this section we will build a web demo to classify images using Gradio. We will be able to build the whole web application in Python, and it will look like this" ], "metadata": { "id": "mg_eEi4luu9R" } }, { "cell_type": "markdown", "source": [ "### Setting up the Image Classification Model" ], "metadata": { "id": "zXerdxtzxKOB" } }, { "cell_type": "markdown", "source": [ "First, we will need an image classification model. For this tutorial, we will use a pretrained Mobile Net model, as it is easily downloadable from Keras. You can use a different pretrained model or train your own." ], "metadata": { "id": "X6jU4ZkUxNJB" } }, { "cell_type": "code", "source": [ "!wget https://hf.space/embed/abidlabs/keras-image-classifier/file/banana.jpg\n", "!wget https://hf.space/embed/abidlabs/keras-image-classifier/file/car.jpg" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "00anQWulwYlR", "outputId": "38946488-66f3-4e74-fe69-00d25d0ac1b6" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "--2022-05-15 08:07:09-- https://hf.space/embed/abidlabs/keras-image-classifier/file/banana.jpg\n", "Resolving hf.space (hf.space)... 54.87.75.213, 18.210.209.118, 34.206.39.3, ...\n", "Connecting to hf.space (hf.space)|54.87.75.213|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 28437 (28K) [image/jpeg]\n", "Saving to: ‘banana.jpg’\n", "\n", "\rbanana.jpg 0%[ ] 0 --.-KB/s \rbanana.jpg 100%[===================>] 27.77K --.-KB/s in 0s \n", "\n", "2022-05-15 08:07:10 (240 MB/s) - ‘banana.jpg’ saved [28437/28437]\n", "\n", "--2022-05-15 08:07:10-- https://hf.space/embed/abidlabs/keras-image-classifier/file/car.jpg\n", "Resolving hf.space (hf.space)... 54.87.75.213, 18.210.209.118, 34.206.39.3, ...\n", "Connecting to hf.space (hf.space)|54.87.75.213|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 79626 (78K) [image/jpeg]\n", "Saving to: ‘car.jpg’\n", "\n", "car.jpg 100%[===================>] 77.76K --.-KB/s in 0.02s \n", "\n", "2022-05-15 08:07:10 (3.84 MB/s) - ‘car.jpg’ saved [79626/79626]\n", "\n" ] } ] }, { "cell_type": "code", "source": [ "inception_net = tf.keras.applications.MobileNetV2()" ], "metadata": { "id": "EI-LD2DUORgC", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "26e56658-b884-4e75-9118-8e5cddac4655" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224.h5\n", "14540800/14536120 [==============================] - 0s 0us/step\n", "14548992/14536120 [==============================] - 0s 0us/step\n" ] } ] }, { "cell_type": "markdown", "source": [ "### Defining a predict function" ], "metadata": { "id": "MYWY_7oExPd5" } }, { "cell_type": "markdown", "source": [ "Next, we will need to define a function that takes in the user input, which in this case is an image, and returns the prediction. The prediction should be returned as a dictionary whose keys are class name and values are confidence probabilities. We will load the class names from [this text file](https://raw.githubusercontent.com/gradio-app/mobilenet-example/master/labels.txt).\n", "\n", "In the case of our pretrained model, it will look like this:" ], "metadata": { "id": "yVHc0kSuxQuC" } }, { "cell_type": "code", "source": [ "# Download human-readable labels for ImageNet.\n", "response = requests.get(\"https://git.io/JJkYN\")\n", "labels = response.text.split(\"\\n\")\n", "\n", "def classify_image(inp):\n", " inp = inp.reshape((-1, 224, 224, 3))\n", " inp = tf.keras.applications.mobilenet_v2.preprocess_input(inp)\n", " prediction = inception_net.predict(inp).flatten()\n", " confidences = {labels[i]: float(prediction[i]) for i in range(1000)}\n", " return confidences" ], "metadata": { "id": "RO2OSENMvSkR" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Let's break this down. The function takes one parameter:\n", "* `inp`: the input image as a numpy array\n", "\n", "Then, the function adds a batch dimension, passes it through the model, and returns:\n", "* `confidences`: the predictions, as a dictionary whose keys are class labels and whose values are confidence probabilities" ], "metadata": { "id": "gGFmr0vMxbX6" } }, { "cell_type": "markdown", "source": [ "### Creating a Gradio Interface" ], "metadata": { "id": "qDONv9Gkxjnh" } }, { "cell_type": "markdown", "source": [ "Now that we have our predictive function set up, we can create a Gradio Interface around it. In this case, the input component is a drag-and-drop image component. To create this input, we can use the `gradio.inputs.Image` class, which creates the component and handles the preprocessing to convert that to a numpy array. We will instantiate the class with a parameter that automatically preprocesses the input image to be 224 pixels by 224 pixels, which is the size that MobileNet expects.\n", "\n", "The output component will be a \"label\", which displays the top labels in a nice form. Since we don't want to show all 1,000 class labels, we will customize it to show only the top 3 images.\n", "\n", "Finally, we'll add one more parameter, the examples, which allows us to prepopulate our interfaces with a few predefined examples. The code for Gradio looks like this:" ], "metadata": { "id": "nsem8PFexno5" } }, { "cell_type": "code", "source": [ "gr.Interface(fn=classify_image, \n", " inputs=gr.inputs.Image(shape=(224, 224)),\n", " outputs=gr.outputs.Label(num_top_classes=3),\n", " examples=[\"banana.jpg\", \"car.jpg\"]).launch(server_port=8050)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 643 }, "id": "ikag2HB1vWWJ", "outputId": "79922b85-1d13-4cdd-d34d-2832d072dace" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Colab notebook detected. To show errors in colab notebook, set `debug=True` in `launch()`\n", "Running on public URL: https://57451.gradio.app\n", "\n", "This share link expires in 72 hours. For free permanent hosting, check out Spaces (https://huggingface.co/spaces)\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", " \n", " " ] }, "metadata": {} }, { "output_type": "execute_result", "data": { "text/plain": [ "(,\n", " 'http://127.0.0.1:8050/',\n", " 'https://57451.gradio.app')" ] }, "metadata": {}, "execution_count": 75 } ] }, { "cell_type": "markdown", "source": [ "Gradio automatically produces sharable links with others, but you can also access the web app with our port as follows:" ], "metadata": { "id": "3vtNvjJ4x1Ny" } }, { "cell_type": "code", "source": [ "public_url" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "XddIL9oaxFGC", "outputId": "ad35ebf7-641d-46dd-d404-ad2ab127d34a" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " \"http://localhost:8050\">" ] }, "metadata": {}, "execution_count": 76 } ] }, { "cell_type": "markdown", "source": [ "For more information, please refer to https://github.com/gradio-app/gradio" ], "metadata": { "id": "6OJN8KHSyFsR" } }, { "cell_type": "markdown", "source": [ "## Deploy web base applocation using Tensorflow.js" ], "metadata": { "id": "Ra51x79aRCfQ" } }, { "cell_type": "markdown", "source": [ "Tensorflow.js is a WebGL accelerated JavaScript library for training and deploying ML models. The TensorFlow.js project includes a `tensorflowjs_converter` tool that can convert a TensorFlow SavedModel or a Keras model file to the TensorFlow.js Layers format: this is a directory\n", "containing a set of sharded weight files in binary format and a model.json file that describes the model’s architecture and links to the weight files. This format is optimized to be downloaded efficiently on the web.\n", "\n", "Users can then download the model and run predictions in the browser using the TensorFlow.js library. Here is a code snippet to give you an idea of what the JavaScript API looks like:\n", "\n", "\n", "```\n", "import * as tf from '@tensorflow/tfjs';\n", "const model = await tf.loadLayersModel('https://example.com/tfjs/model.json');\n", "const image = tf.fromPixels(webcamElement);\n", "const prediction = model.predict(image);\n", "```" ], "metadata": { "id": "Htbhi0Iey96K" } }, { "cell_type": "markdown", "source": [ "For more information, please refer to https://github.com/tensorflow/tfjs." ], "metadata": { "id": "01GAbm5jyifw" } }, { "cell_type": "markdown", "source": [ "## Deploy mobile applocation using Tensorflow Lite" ], "metadata": { "id": "VRiSAp-DRHg6" } }, { "cell_type": "markdown", "source": [ "Once again, doing justice to this topic would require a whole book. If you want to learn more about TensorFlow Lite, check out the O’Reilly book [Practical Deep Learning for Cloud, Mobile, and Edge](https://www.oreilly.com/library/view/practical-deep-learning/9781492034858/) or refer to https://www.tensorflow.org/lite." ], "metadata": { "id": "55tvq75Zzdmi" } }, { "cell_type": "markdown", "source": [ "## Monitoring shift with evidently" ], "metadata": { "id": "VW_L5gzOQ-8Z" } }, { "cell_type": "markdown", "source": [ "### The task at hand: bike demand forecasting" ], "metadata": { "id": "eVVQCG8F2sg6" } }, { "cell_type": "markdown", "source": [ "We took a Kaggle dataset on [Bike Sharing Demand](https://www.kaggle.com/c/bike-sharing-demand/data). Our goal is to predict the volume of bike rentals on an hourly basis. To do that, we have some data about the season, weather, and day of the week." ], "metadata": { "id": "tdeMSRUU2wpx" } }, { "cell_type": "code", "source": [ "content = requests.get(\"https://archive.ics.uci.edu/ml/machine-learning-databases/00275/Bike-Sharing-Dataset.zip\").content\n", "with zipfile.ZipFile(io.BytesIO(content)) as arc:\n", " raw_data = pd.read_csv(arc.open(\"hour.csv\"), header=0, sep=',', parse_dates=['dteday']) " ], "metadata": { "id": "1qTX9Ty1RA88" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "raw_data.head()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "id": "lNPui-BH2_i3", "outputId": "e39e6630-9abe-475f-9a2d-176544113e2b" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " instant dteday season yr mnth hr holiday weekday workingday \\\n", "0 1 2011-01-01 1 0 1 0 0 6 0 \n", "1 2 2011-01-01 1 0 1 1 0 6 0 \n", "2 3 2011-01-01 1 0 1 2 0 6 0 \n", "3 4 2011-01-01 1 0 1 3 0 6 0 \n", "4 5 2011-01-01 1 0 1 4 0 6 0 \n", "\n", " weathersit temp atemp hum windspeed casual registered cnt \n", "0 1 0.24 0.2879 0.81 0.0 3 13 16 \n", "1 1 0.22 0.2727 0.80 0.0 8 32 40 \n", "2 1 0.22 0.2727 0.80 0.0 5 27 32 \n", "3 1 0.24 0.2879 0.75 0.0 3 10 13 \n", "4 1 0.24 0.2879 0.75 0.0 0 1 1 " ], "text/html": [ "\n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
instantdtedayseasonyrmnthhrholidayweekdayworkingdayweathersittempatemphumwindspeedcasualregisteredcnt
012011-01-01101006010.240.28790.810.031316
122011-01-01101106010.220.27270.800.083240
232011-01-01101206010.220.27270.800.052732
342011-01-01101306010.240.28790.750.031013
452011-01-01101406010.240.28790.750.0011
\n", "
\n", " \n", " \n", " \n", "\n", " \n", "
\n", "
\n", " " ] }, "metadata": {}, "execution_count": 24 } ] }, { "cell_type": "markdown", "source": [ "### Train a model" ], "metadata": { "id": "JanaILJc3COD" } }, { "cell_type": "markdown", "source": [ "We trained a random forest model using data for the four weeks from January. Let's imagine that in practice, we just started the data collection, and that was all the data available. The performance of the trained model looked acceptable, so we decided to give it a go.\n", "\n", "We further assume that we only learn the ground truth (the actual demand) **at the end of each week.** That is a realistic assumption in real-world machine learning. Integrating and updating different data sources is not always straightforward. Even after the actual event has occurred! Maybe the daily usage data is stored locally and is only sent and merged in the database once per week." ], "metadata": { "id": "uZHcdMIa3GOS" } }, { "cell_type": "markdown", "source": [ "To run it, we prepare our performance data as a Pandas DataFrame. It should include:\n", "* **Model application logs**—features that went into the model and corresponding prediction; and\n", "* **Ground truth data**—the actual number of bikes rented each hour as our \"target.\"" ], "metadata": { "id": "VQKN3CCfm1Ln" } }, { "cell_type": "code", "source": [ "raw_data.index = raw_data.apply(lambda row: datetime.datetime.combine(row.dteday.date(), datetime.time(row.hr)),\n", " axis=1)" ], "metadata": { "id": "EEvpcEg13Apf" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "raw_data.head()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "id": "KBU02QRbl762", "outputId": "bbd802ce-8650-4cf6-cbb8-a8c702e0afa8" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " instant dteday season yr mnth hr holiday \\\n", "2011-01-01 00:00:00 1 2011-01-01 1 0 1 0 0 \n", "2011-01-01 01:00:00 2 2011-01-01 1 0 1 1 0 \n", "2011-01-01 02:00:00 3 2011-01-01 1 0 1 2 0 \n", "2011-01-01 03:00:00 4 2011-01-01 1 0 1 3 0 \n", "2011-01-01 04:00:00 5 2011-01-01 1 0 1 4 0 \n", "\n", " weekday workingday weathersit temp atemp hum \\\n", "2011-01-01 00:00:00 6 0 1 0.24 0.2879 0.81 \n", "2011-01-01 01:00:00 6 0 1 0.22 0.2727 0.80 \n", "2011-01-01 02:00:00 6 0 1 0.22 0.2727 0.80 \n", "2011-01-01 03:00:00 6 0 1 0.24 0.2879 0.75 \n", "2011-01-01 04:00:00 6 0 1 0.24 0.2879 0.75 \n", "\n", " windspeed casual registered cnt \n", "2011-01-01 00:00:00 0.0 3 13 16 \n", "2011-01-01 01:00:00 0.0 8 32 40 \n", "2011-01-01 02:00:00 0.0 5 27 32 \n", "2011-01-01 03:00:00 0.0 3 10 13 \n", "2011-01-01 04:00:00 0.0 0 1 1 " ], "text/html": [ "\n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
instantdtedayseasonyrmnthhrholidayweekdayworkingdayweathersittempatemphumwindspeedcasualregisteredcnt
2011-01-01 00:00:0012011-01-01101006010.240.28790.810.031316
2011-01-01 01:00:0022011-01-01101106010.220.27270.800.083240
2011-01-01 02:00:0032011-01-01101206010.220.27270.800.052732
2011-01-01 03:00:0042011-01-01101306010.240.28790.750.031013
2011-01-01 04:00:0052011-01-01101406010.240.28790.750.0011
\n", "
\n", " \n", " \n", " \n", "\n", " \n", "
\n", "
\n", " " ] }, "metadata": {}, "execution_count": 26 } ] }, { "cell_type": "code", "source": [ "reference = raw_data.loc['2011-01-01 00:00:00':'2011-01-28 23:00:00']\n", "\n", "target = 'cnt'\n", "prediction = 'prediction'\n", "numerical_features = ['temp', 'atemp', 'hum', 'windspeed', 'hr', 'weekday']\n", "categorical_features = ['season', 'holiday', 'workingday']" ], "metadata": { "id": "9kzjKdUd3M6v" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "reference[numerical_features + categorical_features].shape" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Wic2e-nXtX3e", "outputId": "959deb3e-c827-4b32-a0b4-10230aad08d2" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(618, 9)" ] }, "metadata": {}, "execution_count": 28 } ] }, { "cell_type": "code", "source": [ "regressor = ensemble.RandomForestRegressor(random_state = 0, n_estimators = 50)" ], "metadata": { "id": "h27i8bmk3Pgp" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "regressor.fit(reference[numerical_features + categorical_features], reference[target])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4Oroixuy3RNg", "outputId": "8daf7b4a-5794-446f-cf6d-e9a5768cf56e" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "RandomForestRegressor(n_estimators=50, random_state=0)" ] }, "metadata": {}, "execution_count": 30 } ] }, { "cell_type": "code", "source": [ "ref_prediction = regressor.predict(reference[numerical_features + categorical_features])" ], "metadata": { "id": "caX_rP9i3Z9A" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "reference['prediction'] = ref_prediction" ], "metadata": { "id": "2GEMgW4_mFKX" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Once we trained a model, we take our training dataset and predictions and specify it as the \"Reference\" data. By default, Evidently uses the index as an x-axis in plots. In this case, it is datetime, so we do not add anything explicitly. Otherwise, we would have to specify it in our column mapping." ], "metadata": { "id": "1l5XKuVunKPJ" } }, { "cell_type": "code", "source": [ "column_mapping = ColumnMapping()\n", "\n", "column_mapping.target = target\n", "column_mapping.prediction = prediction\n", "column_mapping.numerical_features = numerical_features\n", "column_mapping.categorical_features = categorical_features" ], "metadata": { "id": "Y1m9tO5imGp5" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Next, we call a corresponding report for regression models." ], "metadata": { "id": "L00A7MrPnwoF" } }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard = Dashboard(tabs=[RegressionPerformanceTab()])\n", "regression_perfomance_dashboard.calculate(reference, None, column_mapping=column_mapping)" ], "metadata": { "id": "6L4hN2avmJxv" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "And display the results right in the Jupyter notebook." ], "metadata": { "id": "0KpXFbben4nH" } }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "025_Z2s7nzSR", "outputId": "a5f15216-5de2-46c9-f1ce-ca2a27357e50" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 35 } ] }, { "cell_type": "markdown", "source": [ "We also save it as a .html file to be able to share it easily." ], "metadata": { "id": "hqrrGQ19n-YR" } }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.save('regression_performance_at_training.html')" ], "metadata": { "id": "UlLhOT-bn5lk" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "We can see that the model has a fine quality given that we only trained on four weeks of data! The error is symmetric and distributed around zero. There is no obvious under- or over-estimation." ], "metadata": { "id": "PoWgZ4pTpkyB" } }, { "cell_type": "markdown", "source": [ "We will continue treating this dataset from model performance in training as our \"reference.\" It gives us a good feel of the quality we can expect from our model in production use. So, we can contrast the future performance against this benchmark." ], "metadata": { "id": "nE4AEQjDptqg" } }, { "cell_type": "markdown", "source": [ "### The first week in production" ], "metadata": { "id": "A0bri5iupxKw" } }, { "cell_type": "markdown", "source": [ "Observing the model in production has straightforward goals. We want to detect if something goes wrong. Ideally, in advance. We also want to diagnose the root cause and get a quick understanding of how to address it. Maybe, the model degrades too fast, and we need to retrain it more often? Perhaps, the error is too high, and we need to adapt the model and rebuild it? Which new patterns are emerging?" ], "metadata": { "id": "VLyYWGkGp5Qf" } }, { "cell_type": "markdown", "source": [ "**In our case, we simply start by checking how well the model performs outside the training data.** Our first week becomes what would have otherwise been a holdout dataset." ], "metadata": { "id": "BsMCe5LtqCA_" } }, { "cell_type": "markdown", "source": [ "For demonstration purposes, we generated all predictions for several weeks ahead in a single batch. In reality, we would run the model sequentially as the data comes in." ], "metadata": { "id": "dQxmp_hcqcGO" } }, { "cell_type": "code", "source": [ "production = raw_data.loc['2011-01-29 00:00:00':'2011-02-28 23:00:00']\n", "production_prediction = regressor.predict(production[numerical_features + categorical_features])\n", "production['prediction'] = production_prediction" ], "metadata": { "id": "VBj1VwxhqXCv" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Let's start by comparing the performance in the first week to what we have seen in training. The first 28 days are our Reference dataset; the next 7 are the Production." ], "metadata": { "id": "QJZfR_1nqN9v" } }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.calculate(reference, production.loc['2011-01-29 00:00:00':'2011-02-07 23:00:00'], column_mapping=column_mapping)" ], "metadata": { "id": "9NWPkfgFoA2E" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "q1uGuqtEqf0K", "outputId": "f1c36b96-a04b-4547-9967-ff10f2acb208" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 39 } ] }, { "cell_type": "markdown", "source": [ "The error has slightly increased and is leaning towards underestimation. Let's check if there is any statistical change in our target. To do that, we will generate the Target Drift report." ], "metadata": { "id": "qbkfM_41qzyg" } }, { "cell_type": "code", "source": [ "# We call the report from the Evidently tabs. In the regression model, our target is numerical, so we pick a matching report\n", "target_drift_dashboard = Dashboard(tabs=[NumTargetDriftTab()])\n", "target_drift_dashboard.calculate(reference, production.loc['2011-01-29 00:00:00':'2011-02-07 23:00:00'], \n", " column_mapping=column_mapping)" ], "metadata": { "id": "wVXKswZzqkHG" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "NumTargetDriftTab.list_widgets()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "uUbns4B12-Cx", "outputId": "20a14d5a-dfaf-455a-a0cb-367f42bee52e" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['Target Drift',\n", " 'Target Correlations',\n", " 'Target Values',\n", " 'Prediction Drift',\n", " 'Prediction Correlations',\n", " 'Prediction Values',\n", " 'Target (Prediction) Behavior By Feature']" ] }, "metadata": {}, "execution_count": 55 } ] }, { "cell_type": "code", "source": [ "target_drift_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "ekF8Wn3Gq4lH", "outputId": "d3f040a5-69f0-4f71-9116-2b9b2a6e4a37" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 56 } ] }, { "cell_type": "markdown", "source": [ "We can see that the distribution of the actual number of bikes rented remains sufficiently similar. To be more precise, the similarity hypothesis is not rejected. No drift is detected. The distributions of our predictions did not change much either.\n", "\n", "Despite this, a rational decision is to update your model by including the new week's data. This way, the model can continue to learn, and we can probably improve the error. For the sake of demonstration, we'll stick to see how fast things go really wrong." ], "metadata": { "id": "LWS-ySJzvECQ" } }, { "cell_type": "markdown", "source": [ "### The second week: failing to keep up" ], "metadata": { "id": "4QDm3J2ivVzI" } }, { "cell_type": "markdown", "source": [ "Once again, we benchmark our new week against the reference dataset." ], "metadata": { "id": "DLzuOsh8vbdB" } }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.calculate(reference, production.loc['2011-02-07 00:00:00':'2011-02-14 23:00:00'], \n", " column_mapping=column_mapping)" ], "metadata": { "id": "6dEvt5nSq6KS" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "s2_v4IwHvl2n", "outputId": "ec1cc177-1a64-4479-bc86-0f7a2bd4e77e" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 43 } ] }, { "cell_type": "markdown", "source": [ "At first glance, the model performance in the second week does not differ much. MAE remains almost the same. But, the skew towards under-estimation continues to grow. It seems that the error is not random! To know more, we move to the plots. We can see that the model catches overall daily trends just fine. So it learned something useful! **But, at peak hours, the actual demand tends to be higher than predicted.**\n", "\n", "In the error distribution plot, we can see how it became \"wider,\" as we have more predictions with a high error. The shift to the left is visible, too. In some extreme instances, we have errors between 80 and 40 bikes that were unseen previously." ], "metadata": { "id": "MolwD2Jmz8ep" } }, { "cell_type": "markdown", "source": [ "Let's check our target as well." ], "metadata": { "id": "_x9MCofe0lMp" } }, { "cell_type": "code", "source": [ "target_drift_dashboard.calculate(reference, production.loc['2011-02-07 00:00:00':'2011-02-14 23:00:00'], \n", " column_mapping=column_mapping)" ], "metadata": { "id": "uGXfwnbQvnsP" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "target_drift_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "0I5SNTQV0nOi", "outputId": "bf7f7b61-007e-42f4-8266-8763d2776a03" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 45 } ] }, { "cell_type": "code", "source": [ "target_drift_dashboard.save('target_drift_after_week2.html')" ], "metadata": { "id": "PncDmoVd3a3R" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Things are getting interesting!\n", "\n", "**We can see that the target distribution is now different: the similarity hypothesis is rejected. Literally, people are renting more bikes. And this is a statistically different change from our training period.**" ], "metadata": { "id": "CbOFix3I1Leh" } }, { "cell_type": "markdown", "source": [ "But, the distribution of our predictions does not keep up! That is an obvious example of **model decay. Something new happens in the world, but it misses the patterns.**" ], "metadata": { "id": "m2fLq7_B1a-C" } }, { "cell_type": "markdown", "source": [ "It is tempting to investigate further. Is there anything in the data that can explain this change? If there is some new signal, retraining would likely help the model to keep up. Maybe, it would pick up these patterns in retraining. But for now, we simply move on to the next week without any updates." ], "metadata": { "id": "wXe-5sUn1yWJ" } }, { "cell_type": "markdown", "source": [ "### Week 3: when things go south" ], "metadata": { "id": "RfIqni0-4hBS" } }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.calculate(reference, production.loc['2011-02-15 00:00:00':'2011-02-21 23:00:00'], \n", " column_mapping=column_mapping)" ], "metadata": { "id": "J61Zn7jd1Ope" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "regression_perfomance_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "89PUMmNO4qCF", "outputId": "b055491b-e8b3-4cb7-a0df-67e2a7b36e04" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 58 } ] }, { "cell_type": "markdown", "source": [ "Okay, now things do look bad. On week 3, we face a major quality drop. Both absolute and percentage error grew significantly. If we look at the plots, the model predictions are visibly scattered. We also face a n**ew data segment with high demand that the model fails to predict.** But even within the known range of target value, the model now makes errors. Things did change since the training. We can see that the model does not extrapolate well. The predicted demand stays within the same known range, while actual values are peaking." ], "metadata": { "id": "jLXpBPTX4yFh" } }, { "cell_type": "markdown", "source": [ "If we zoom in on specific days, we might suggest that the error is higher on specific (active) hours of the day. We are doing just fine from 10 pm to 6 am! We have a high error, and it has a clear skew towards underestimation." ], "metadata": { "id": "rnehcGDO5MvQ" } }, { "cell_type": "markdown", "source": [ "In our example, we particularly want to understand the segment where the model underestimates the target function. The `Error Bias table` gives up more details. We sort it by the `\"Range%\"` field. If the values of a specific feature are significantly different in the group where the model under- or over-estimates, this feature will rank high. **In our case, we can see that the extreme errors are dependent on the \"temp\" (temperature) and \"atemp\" (feels-like temperature) features.**" ], "metadata": { "id": "M2WaWSyu5kJ6" } }, { "cell_type": "markdown", "source": [ "After this quick analysis, we have a more specific idea about model performance and its weaknesses. The model faces new, unusually high demand. Given how it was trained, it tends to underestimate it. On top of it, these errors are not at all random. At the very least, they are related to the temperature we observe. The higher it is, the larger the underestimation. **It suggests new patterns that are related to the weather that the model could not learn before. Days got warmer, and the model went rogue.**" ], "metadata": { "id": "mY2qSYOB65dQ" } }, { "cell_type": "markdown", "source": [ "We should retrain as soon as possible and do this often until we learn all the patterns. If we are not comfortable with frequent retraining, we might choose an algorithm that is more suitable for time series or is better in extrapolation." ], "metadata": { "id": "IM8Tmqf07BuO" } }, { "cell_type": "markdown", "source": [ "### Data Drift" ], "metadata": { "id": "sh-nc7Qp7NKK" } }, { "cell_type": "markdown", "source": [ "In practice, once we receive the ground truth, we can indeed course-correct quickly. Had we retrained the model after week one, it would have likely ended less dramatically. **But what if we do not have the ground truth available? Can we catch such decay in advance?**\n", "\n", "In this case, we can analyze the data drift. We do not need actuals to calculate the error. Instead, our goal is to see if the input data has changed." ], "metadata": { "id": "g6WPMeSu7O2K" } }, { "cell_type": "markdown", "source": [ "Once again, let's compare the first week of production to our data in training. We can, of course, look at all our features. But we can also conclude that categorical features (like \"season,\" \"holiday\" and \"workingday\") are not likely to change. Let's look at numerical features only!\n", "\n", "We specify these features so that the tool applies the correct statistical test. It would be Kolmogorov-Smirnov in this case." ], "metadata": { "id": "Ib0cUpnS7ixR" } }, { "cell_type": "code", "source": [ "column_mapping = ColumnMapping()\n", "column_mapping.numerical_features = numerical_features" ], "metadata": { "id": "Fgg-JXN24rrd" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "data_drift_dashboard = Dashboard(tabs=[DataDriftTab()])\n", "data_drift_dashboard.calculate(reference, production.loc['2011-01-29 00:00:00':'2011-02-07 23:00:00'], \n", " column_mapping=column_mapping)" ], "metadata": { "id": "qDk5S6e-8Q8-" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "data_drift_dashboard.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "1IPkktjN8RZ-", "outputId": "215766be-e75f-43c5-c185-8331d215887c" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Loading...
\n", "\n", "\n", "\n" ] }, "metadata": {}, "execution_count": 62 } ] }, { "cell_type": "markdown", "source": [ "Once we show the report, it returns an answer. We can see already during the first week there is a statistical change in feature distributions.\n", "\n", "Let's zoom in on our usual suspect—temperature. The report gives us two views on how the feature distributions evolve with time. We can notice how the observed temperature becomes higher day by day. The values clearly drift out of our green corridor (one standard deviation from the mean) that we saw in training. Looking at the steady growth, we can suspect an upward trend." ], "metadata": { "id": "uaLxHzFP8gR6" } }, { "cell_type": "markdown", "source": [ "As we checked earlier, we did not detect drift in the model predictions after week one. Given that our model is not good at extrapolating, we should not really expect it. Such prediction drift might still happen and signal about things like broken input data. Otherwise, we would observe it if we had a more sensitive model. Regardless of this, the data drift alone provides excellent early monitoring to detect the change and react to it." ], "metadata": { "id": "uBklQ9iN86dz" } }, { "cell_type": "markdown", "source": [ "For more information please refer to https://github.com/evidentlyai/evidently, https://github.com/SeldonIO/alibi-detect, https://github.com/great-expectations/great_expectations or https://github.com/whylabs/whylogs" ], "metadata": { "id": "0rSXNKI_9QtJ" } }, { "cell_type": "code", "source": [ "" ], "metadata": { "id": "MLhDp66e8VfP" }, "execution_count": null, "outputs": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" }, "nav_menu": {}, "toc": { "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 6, "toc_cell": false, "toc_section_display": "block", "toc_window_display": false }, "colab": { "name": "13_Deploy.ipynb", "provenance": [], "collapsed_sections": [], "toc_visible": true }, "accelerator": "GPU" }, "nbformat": 4, "nbformat_minor": 0 }